[Endpoint][EPM] Retrieve Index Pattern from Ingest Manager #63016

jonathan-buttner · 2020-04-08T19:55:29Z

Important Change

When having the endpoint send metadata use the index metrics-endpoint-default-1.

Summary

~~I'm going to leave this PR in draft until the the endpoint-ingest dependency PR is merged here: #62871~~

~~Once that PR is merged this one's number of changed files will go down.~~

This PR exposes functionality in the Ingest Manager to construct an index pattern based on the contents of a package. The endpoint app will rely on the ingest manager to retrieve index patterns for querying elasticsearch. The advantage of using the package to construct the index pattern is it isolates the logic for defining the index patterns to only the endpoint package so it only needs to be maintained there. The downside is a request must be made to access a saved object when the index pattern is retrieved for each time that index pattern is needed. This is done once per request when needed. If a route handler makes multiple queries, it only retrieves the pattern once. I spoke with the Kibana team about the extra overhead of using saved objects and they recommended not worrying about it for now. If it becomes an issue we can implement a cache within the endpoint app.

Notable Changes

Ingest Manager

Modified the structure of the saved objects used to store information about installed packages
- I added a patterns field which is a map of dataset to constructed index pattern (e.g. events -> events-endpoint-*)
Added a setup contract that allows other plugins server to make calls into the ingest manager's server without using an http handler
Created a helper service in the api_integration test directory to handle setting up the ingest manager before the endpoint api_integration tests. This is needed because even though the endpoint app depends on the ingest manager, since the UIs are not used it's setup function will not be called

Endpoint

Removed almost all references to hardcoded index patterns, there is still one place within the alert handlers that remains because we need to encode the index of alerts in the id field when returning them to the UI. I'll make that change in a follow up PR once this merged.
Added the ingest manager as a dependency for the endpoint app
Added a class for communicating with the ingest manager
Added a API route for retrieving an index pattern so the alert list view can set the index pattern accordingly
Added api_integration tests for the scenario when the ingest manager is not setup

Testing

If you'd like to test this PR you'll need to add a couple flags to your kibana.dev.yml file:

xpack.ingestManager.enabled: true
#xpack.ingestManager.epm.registryUrl: 'http://127.0.0.1:8080'
xpack.ingestManager.epm.enabled: true
xpack.ingestManager.fleet.enabled: true

The ingest manager will pull the packages from the default package registry URL. If you did want to run your own package registry you can point ingest manager to it using the xpack.ingestManager.epm.registryUrl (more info here: https://github.com/elastic/package-registry and https://github.com/elastic/kibana/blob/master/x-pack/plugins/ingest_manager/README.md)

For the endpoint app you can try populating elasticsearch with events from the alerts es_archive and make sure you can still see them in the alert list view:

…enabling ingest in the base tests

…st-setup

elasticmachine · 2020-04-08T19:55:32Z

Pinging @elastic/endpoint-app-team (Feature:Endpoint)

elasticmachine · 2020-04-08T19:55:33Z

Pinging @elastic/ingest-management (Feature:EPM)

x-pack/plugins/endpoint/server/plugin.test.ts

paul-tavares

LGTM 👍

paul-tavares · 2020-04-15T20:05:41Z

x-pack/plugins/endpoint/server/index_pattern.ts

+
+  async getIndexPattern(ctx: RequestHandlerContext, datasetPath: string) {
+    try {
+      const pattern = await this.service.getESIndexPattern(


Should (can?) any of these call be cached?

Yeah, I haven't created an issue for it yet but we'll create a cache so we don't have to do so many calls to the saved objects.

Issue: https://github.com/elastic/endpoint-app-team/issues/340

paul-tavares · 2020-04-15T20:09:42Z

x-pack/test/endpoint_api_integration_no_ingest/ftr_provider_context.d.ts

@@ -0,0 +1,11 @@
+/*


Just curious why you had to create this typescript definition file.

Yeah it wasn't necessary, it seemed like most test directories had their own but I think it's only useful if we define our own services and page_objects.

charlie-pichette

Changes for tests look good. 👍

oatkiller · 2020-04-16T14:05:15Z

Created a helper service in the api_integration test directory to handle setting up the ingest manager before the endpoint api_integration tests. This is needed because even though the endpoint app depends on the ingest manager, since the UIs are not used it's setup function will not be called

@jonathan-buttner Can you explain this a little? The Endpoint client code is responsible for calling the ingest server side setup function? I know you mentioned it to me, but I can't remember what the reasoning was.

x-pack/plugins/endpoint/scripts/resolver_generator.ts

x-pack/plugins/endpoint/server/index_pattern.ts

oatkiller · 2020-04-16T14:14:01Z

x-pack/plugins/endpoint/server/index_pattern.ts

+    return await this.getIndexPattern(ctx, EndpointAppConstants.EVENT_DATASET);
+  }
+
+  async getMetadataIndexPattern(ctx: RequestHandlerContext) {


would you mind adding a quick doc comment to public methods? maybe

/** * fetch and return the index pattern used to store Endpoint host metadata. */

I think these types of comments will help a lot, esp. considering the size of the org

x-pack/plugins/endpoint/server/mocks.ts

x-pack/plugins/endpoint/server/routes/alerts/details/handlers.ts

x-pack/plugins/endpoint/server/routes/alerts/list/lib/index.ts

oatkiller · 2020-04-16T14:20:52Z

x-pack/plugins/endpoint/server/routes/index_pattern.ts

+  };
+}
+
+export function registerIndexPatternRoute(router: IRouter, endpointAppContext: EndpointAppContext) {


is this for the alert list feature? if so, thoughts on moving it to the alerting routes module?

Yeah it's currently only used for the alert list. The alert list uses an index pattern when doing time based filtering I think. I think having an index pattern is required when doing that.

I could foresee other pages (Host maybe?) needing an index pattern if they want to do any time based filtering too. If you'd prefer it in the alert list for now until it's needed in the future, that's fine too though.

x-pack/plugins/endpoint/server/routes/metadata/metadata.test.ts

oatkiller · 2020-04-16T14:24:40Z

x-pack/plugins/endpoint/server/routes/metadata/query_builders.test.ts

@@ -34,7 +39,7 @@ describe('query builder', () => {
          aggs: {
            total: {
              cardinality: {
-                field: 'host.id.keyword',
+                field: 'host.id',


im a bit of a noob about this. can you explain this change? is this just keeping us in sync w/ latest mappings?

Yeah it's because the mapping that was being used for metadata was based off a dynamically generated mapping created by default by elasticsearch when no mapping exists. Now that we have solidified and generated a mapping using ECS we want to use that. With the new mapping the field keyword no longer exists.

https://www.elastic.co/blog/strings-are-dead-long-live-strings

oatkiller · 2020-04-16T14:27:46Z

x-pack/plugins/endpoint/server/test_data/all_metadata_data.json

-    "successful" : 1,
-    "skipped" : 0,
-    "failed" : 0
+  "took": 343,


can you explain the change here? also, anyone know why this is large and not gzipped?

@jonathan-buttner This will be on me to fix.

@oatkiller most of the changes are just spacing and renaming the index being used:

my vs code default formatter is removing the spaces like these, I can revert them if you prefer:

I believe this is not gzipped because it's directly imported in typescript code here:
https://github.com/elastic/kibana/pull/63016/files/2829689ae9674970d57644212863734d34d2cd5b#diff-b1e4abedd6d2153cf169796ebae56a93R27

@nnamdifrankie yeah if you could fix it that'd be great!

oatkiller · 2020-04-16T14:37:09Z

x-pack/plugins/ingest_manager/server/services/es_index_pattern.ts

+  ): Promise<string | undefined>;
+}
+
+export class ESIndexPatternSavedObjectService implements ESIndexPatternService {


side note, classes in ts are also interfaces, so ESIndexPatternSavedObjectService can be used exactly the same as ESIndexPatternService Using a class as an interface

Ah cool! Didn't realize that.

oatkiller · 2020-04-16T14:40:13Z

x-pack/test/api_integration/apis/endpoint/index_pattern.ts

+export default function({ getService }: FtrProviderContext) {
+  const supertest = getService('supertest');
+
+  describe('Endpoint index pattern API', () => {


I mentioned this earlier, but what is the purpose of exposing these APIs?

Yeah it doesn't have to be, it's only used by the alert list but could be useful if other UIs need to do time based filtering of a list.

jonathan-buttner · 2020-04-16T16:04:52Z

Created a helper service in the api_integration test directory to handle setting up the ingest manager before the endpoint api_integration tests. This is needed because even though the endpoint app depends on the ingest manager, since the UIs are not used it's setup function will not be called

@jonathan-buttner Can you explain this a little? The Endpoint client code is responsible for calling the ingest server side setup function? I know you mentioned it to me, but I can't remember what the reasoning was.

@oatkiller yeah, the Ingest Manager plugin will handle calling the setup routine in its front end plugin's start function. This means that initialization occurs when the Kibana UI is loaded (you don't have to open any particular app, just log in to Kibana). The Ingest manager communicates the status of the initialization via the plugin's contract using a boolean. So the Endpoint app does not call setup. For api_integration tests there is no UI to load so the initialization does not occur. The endpoint api_integration tests need the index pattern to query elasticsearch and still need to get it from the Ingest manager but can only retrieve it after the ingest manager is initialized. To force the initialization to occur during api_integration tests, I created a service that handles calling the setup API route: https://github.com/jonathan-buttner/kibana/blob/endpoint-ingest-index-pattern/x-pack/test/api_integration/services/ingest_manager.ts#L22

That service is then used before our endpoint api_integration tests run here: https://github.com/jonathan-buttner/kibana/blob/endpoint-ingest-index-pattern/x-pack/test/api_integration/apis/endpoint/index.ts#L17

kibanamachine · 2020-04-16T22:29:57Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: ec0a649

History

💚 Build #41342 succeeded ec0a649
💚 Build #41074 succeeded 2829689
💔 Build #41020 failed efdcc2d
💚 Build #40994 succeeded dd033d4
💚 Build #40978 succeeded 1aa9388

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: (40 commits) [APM]Upgrade apm-rum agent to latest version to fix full page reload (elastic#63723) add deprecation warning for legacy 3rd party plugins (elastic#62401) Migrate timelion vis (elastic#62819) Replacebad scope link with actual values (elastic#63444) Index pattern management UI -> TypeScript and New Platform Ready (create_index_pattern_wizard) (elastic#63111) [SIEM] Threat hunting enhancements: Filter for/out value, Show top field, Copy to Clipboard, Draggable chart legends (elastic#61207) [Maps] fix term join agg key collision (elastic#63324) [Ingest] Fix agent config key sorting (elastic#63488) [Monitoring] Fixed server response errors (elastic#63181) update elastic charts to 18.3.0 (elastic#63732) Start services (elastic#63720) [APM] Encode spaces when creating ML job (elastic#63683) Uptime 7.7 docs (elastic#62228) [DOCS] Updates remote cluster and ccr docs (elastic#63517) [Maps] Add 3rd party vector tile support (elastic#62084) [Endpoint][EPM] Retrieve Index Pattern from Ingest Manager (elastic#63016) [Endpoint] Host Details Policy Response Panel (elastic#63518) [Uptime] Certificate expiration threshold settings (elastic#63682) Refactor saved object types to use `namespaceType` (elastic#63217) [SIEM][CASE] Create comments sequentially (elastic#63692) ...

…63767) * Endpoint successfully depending on ingest manager to initialize * Moving the endpoint functional tests to their own directory to avoid enabling ingest in the base tests * Removing page objects and other endpoint fields from base functional * Updating code owners with new functional location * Adding index pattern functionality * Missed a file * Pointing resolver tests at endpoint functional tests * Pointing space tests at the endpoint functional directory * Adding ingest service to do setup and tests for 500s * Correcting services path * Adding jest test names * Updating es archives with the correct mapping and index names * Fixing import error * Adding resolver tests to code owners * enabling epm flag for functional tests * adding correct tag to test * Removing the version information and unneeded xsrf * Addressing endpoint index pattern feedback * Removing unused import * Renaming index pattern to es index pattern * Fixing missed index pattern calls * Removing unused import * Fixing type error * Moving es_index_pattern outside of installed and fixing function name * Keeping the event index the same for now * Wrapping index pattern await in try catch * Address PR feedback, adding comments Co-authored-by: Elastic Machine <[email protected]>

jonathan-buttner added 15 commits April 7, 2020 18:08

Endpoint successfully depending on ingest manager to initialize

b69c2ad

Fixing conflicts

7c7eaee

Moving the endpoint functional tests to their own directory to avoid …

c9f0507

…enabling ingest in the base tests

Removing page objects and other endpoint fields from base functional

b82762b

Updating code owners with new functional location

cb45dfe

Adding index pattern functionality

9e4f6e3

Missed a file

8cbfa42

Pointing resolver tests at endpoint functional tests

a1f2bd0

Pointing space tests at the endpoint functional directory

f13231a

Merge branch 'endpoint-ingest-setup' into endpoint-ingest-index-pattern

0c9946d

Adding ingest service to do setup and tests for 500s

0140395

Correcting services path

6c7b7b4

Merge branch 'master' of github.com:elastic/kibana into endpoint-inge…

745c15d

…st-setup

Adding jest test names

661f860

Fixing conflicts

b839152

jonathan-buttner added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project Feature:Endpoint Elastic Endpoint feature v7.8.0 labels Apr 8, 2020

jonathan-buttner requested a review from jfsiii April 8, 2020 19:55

Updating es archives with the correct mapping and index names

cf1a071

nnamdifrankie reviewed Apr 9, 2020

View reviewed changes

x-pack/plugins/endpoint/server/plugin.test.ts Outdated Show resolved Hide resolved

Fixing import error

9ff1e7b

ruflin assigned jonathan-buttner Apr 9, 2020

elastic deleted a comment from kibanamachine Apr 9, 2020

jonathan-buttner added 2 commits April 9, 2020 10:33

Fixing ci test script conflicts

8272634

Adding resolver tests to code owners

3599f46

jonathan-buttner added 3 commits April 15, 2020 11:11

Keeping the event index the same for now

dd033d4

Wrapping index pattern await in try catch

efdcc2d

Fixing merge conflicts

2829689

paul-tavares approved these changes Apr 15, 2020

View reviewed changes

charlie-pichette approved these changes Apr 16, 2020

View reviewed changes